Method and system for extended depth of field calculation for microscopic images

ABSTRACT

The invention relates to an image processing method and system for constructing composite image with extended depth of field. The composite image may be constructed from a plurality of source images of a scene stored in an image stack. The method includes aligning the images in the image stack such that every image in the image stack is aligned with other images in the stack, performing illumination and color correction on the aligned images in the image stack, generating an energy matrix for each pixel of each illumination and color corrected image in the image stack by computing energy content for each pixel, generating a raw index map that contains the location of every pixels having maximum energy level among all the images in the image stack, generating degree of defocus ma and constructing the composite image.

FIELD OF THE INVENTION

The invention relates to image processing field in general andparticularly to a method and system for extending depth of field in animaging device.

BACKGROUND OF THE INVENTION

Limited depth of field is a common problem in imaging devices such asconventional light microscope. Objects imaged in such cases are sharplyin focus over a limited distance known as the depth of field. The Depthof field (DOF) is the distance between the nearest and farthest objectsin a scene that appear acceptably sharp in an image. Typically sharpnessdecreases with depth resulting in blurriness in at least some part ofthe image. Also, in order to capture a large amount of light from asmall specimen under the imaging device, one needs to have a highnumerical aperture. However, high numerical aperture results in a veryshallow depth of field, due to which it is not possible to have allregion of the scene to be in focus.

To improve the depth of field in a captured image, one or more digitalimage processing techniques may be employed. Using the digital imageprocessing techniques, images taken at different depths of field of thesame scene may be combined to produce a single composite image. Thedigital image processing techniques involve capturing multiple images ofthe same scene to form an image stack, identifying focused part frommultiple images in the stack and recreating a single image with betterdepth of field by combining the focused parts. During the digitalprocessing process, index information of the images in the stack iscollected and processed to generate a depth map and composite imageor/and a 3D model of the scene.

Typically, greater the number of images in the image stack greater isthe DOF in the composite image. Though with the increase in number ofimages in the stack, the complexity, time required for processing theimages, errors in the composite image and memory requirement alsoincreases. There are many processing techniques which provide solutionsto improve the depth of field. However, the known solutions have one orthe other drawback such as misalignment of the images in the stack,illumination variations in the composite image, noises in the Depth mapand composite image, low quality of the composite image with blotchybackground, edge shadowing and depth cross over time complexity of theprocesses involved in depth of field calculations too many manuallyconfigurable parameters and unable to manage large image stacks.

Therefore, there is a need to have an improved method and system fordigital image processing that may address at least one of the abovementioned limitations.

SUMMARY OF THE INVENTION

According to embodiments of the invention an image processing method forconstructing a composite image with extended depth of field isdisclosed. The composite image may be constructed from a plurality ofsource images of a scene stored in at least one image stack. Theplurality of source images may be taken at substantially identicalfields of view. The method includes aligning the images in the imagestack such that every image in the image stack is aligned with otherimages in the stack, performing illumination and color correction on thealigned images in the image stack, generating an energy matrix for eachpixel of each illumination and color corrected image in the image stackby computing energy content for each pixel, generating a raw index mapthat contains the location of every pixels having maximum energy levelamong all the images in the image stack, generating degree of defocusmap by comparing the energy content at a particular pixel in all theimages against a reference signal and repeating the process for all thepixels and constructing the composite image using raw index map anddegree of defocus map.

According to another embodiment a system for constructing a compositeimage with extended depth of field, from a plurality of source images ofa scene is disclosed. The disclosed system include a memory for storinga plurality of source images of a scene taken at substantially identicalfields of view, a processing unit for processing the images stored inthe memory, to align the images in the image stack such that every imagein the image stack is aligned with other images in the stack, performillumination and color correction on the aligned images in the imagestack, generate an energy matrix for each pixel of each illumination andcolor corrected image in the image stack by computing energy content foreach pixel, generate a raw index map that contains the location of everypixels having maximum energy level among all the images in the imagestack, generate degree of defocus map by comparing the energy content ata particular pixel against a reference signal and constructing thecomposite image using raw index map and degree of defocus map and anoutput unit for displaying the composite image received from theprocessing unit.

Other aspects, advantages, and salient features of the invention willbecome apparent to those skilled in the art from the following detaileddescription, which, taken in conjunction with the annexed drawings,discloses exemplary embodiments of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainexemplary embodiments of the invention will be more apparent from thefollowing description taken in conjunction with the accompanyingdrawings in which:

FIG. 1 illustrates a flow chart of a method for processing a pluralityof images taken of a scene to generate a composite image with extendeddepth of field, according to one embodiment of the invention;

FIG. 2 illustrates a method for performing illumination and colorcorrection according to an embodiment of the invention; and

FIG. 3 illustrates a block diagram of a system for constructing acomposite image with extended depth of field, from a plurality of sourceimages of a scene according to one embodiment of the invention.

Persons skilled in the art will appreciate that elements in the figuresare illustrated for simplicity and clarity and may have not been drawnto scale. For example, the dimensions of some of the elements in thefigure may be exaggerated relative to other elements to help to improveunderstanding of various exemplary embodiments of the disclosure.

Throughout the drawings, it should be noted that like reference numbersare used to depict the same or similar elements, features, andstructures.

DETAILED DESCRIPTION OF THE INVENTION

The following description with reference to the accompanying drawings isprovided to assist in a comprehensive understanding of exemplaryembodiments of the invention as defined by the claims and theirequivalents. It includes various specific details to assist in thatunderstanding but these are to be regarded as merely exemplary.Accordingly, those of ordinary skill in the art will recognize thatvarious changes and modifications of the embodiments described hereincan be made without departing from the scope and spirit of theinvention. In addition, descriptions of well-known functions andconstructions are omitted for clarity and conciseness.

FIG. 1 illustrates a flow chart of a method 100 for processing aplurality of images taken of a scene to generate a composite image withextended depth of field, according to an exemplary embodiment of theinvention. The extended depth of field indicates a greater depth offield in the composite/processed image as compared to the original imagebefore processing.

At step 102, the method obtains a plurality of image captured from adefined subject, where the images are taken from different positions forsaid subject. According to an embodiment, the images may be capturedfrom any known imaging device such as but not limiting to, opticalmicroscope, digital camera, etc. According to another embodiment, theimages may be obtained from an archive having images of said subjectcaptured from different positions. According to yet another embodiment,the different positions include capturing images from different ‘Z’level of the imaging device. The obtained images are arranged in a stackto form an image stack.

At step 104, the method performs image alignment. The image alignment isthe process of transforming different sets of image data in the imagestack so that all the images may have one coordinate system. The imagedata may be obtained from different sensors, at different time, or fromdifferent viewpoints. The image alignment process enables processing ofthe images obtained from the different measurements/sources. The imagealignment process involves finding an optimal one-to-one mapping betweenthe pixels in one image to those in other images in the stack.

According to one embodiment, a bi-directional image alignment method maybe used for aligning the images in the image stack. The bi-directionalimage aligning method may include arranging the images in a sequence,such that the images are arranged as per their respective distance fromthe ‘Z’ level. Among the image sequence a first reference image may beselected, such that the first reference image is the substantiallycentral image in the image sequence. The alignment process furtherincludes comparing the first reference image with immediate left sideand immediate right side images in the sequence. The geometrictransformation between the immediate left side image and the immediateright side image may be calculated with respect to the first referenceimage. Based on calculations, the immediate left side image and theimmediate right side image may be aligned with the first referenceimage. Subsequently immediate left side image may be compared withsecond image on the left side of the first reference image in the imagesequence and immediate right side image may be compared with the secondimage on the right side of the first reference image in the imagesequence and aligned with the immediate left side and the immediateright side images respectively. The process may be repeated for all theimages thereby resulting in an aligned image stack that hassubstantially all images aligned with each other. The first and lastimage in the stack may have large variations and processing images inone direction that is from first to last may not provide effectivelyaligned image stack. Moreover processing images in one direction mayresult in a lot of time and memory consumption. On the other handdisclosed two way processing of images reduces time and result in betteraligned stack of images.

According to an embodiment of the invention, the process of comparisonand alignment may be performed by any known conventional method in theimage alignment process. According to another embodiment of theinvention, the process of comparison and alignment may be performed bythe parametric image alignment process. The parametric image alignmentprocess includes specifying the particular geometric parametrictransformation and then estimating the parameter by means of any knownoptimization method. The miss-aligned images generally have an affinetransformation and six parameters (a, b, c, d, e, and f) needed to beestimated for this:x′=a*x+b*y+cy′=d*x+e*y+f

These parameters may be identified using any known suitable optimizationmethod. According to an exemplary embodiment of the invention, thedisclosed method uses a hierarchical coarse-to-fine approach ofparametric image alignment using gradient descent optimization methodand normalized cross correlation as cost function based on imagepyramids.

The exemplary illustrated method of comparison and alignment includesreducing the resolution on the images in the image stack by a scalefactor in a range of between ½ to 1/16. According to yet anotherembodiment, the resolution of the images may be reduced by a scalefactor of ⅛ of the original resolution to generate a stack of downsampled images. The method further includes creating an image pyramid ofdown sampled images and performing a coarse estimation of thetransformation on the down sampled images. In microscopic imaging, thewhole series of images or some part of a series of images may undergothrough same transformations. Hence, if the second image undergoesthrough substantially same transformation, the transformation parametersobtained from the first image may be used as a clue or initial guess oftransformation of next image. According to an embodiment, thehierarchical coarse-to-fine parametric image alignment process isimplemented by a guided optimization process. The guided optimizationprocess includes implementing an initial guess method/algorithm tosearch for global optima of the cost function faster by making it tostart the iterations near the global optima position of the previousimage cost function. The illustrated process of comparison and alignmentis only exemplary in nature and may not be construed limiting on theinvention. Any other known method/process of comparison and alignmentmay be used without going beyond the scope of the invention.

At step 106, the method performs illumination and color correction onthe aligned stack of images. FIG. 2 illustrates an exemplary flow chart200 to illustrate the method for performing illumination and colorcorrection according to an embodiment of the invention. RI₁, RI₂, RI₃ .. . RI_(n), refers to the aligned images in the stack of images. Themethod 200 of performing illumination and color correction, at step 202may include selecting at least two consecutive images from the stack ofaligned images, where one of the image is considered as a secondreference image and the other is considered as first sample image. Atstep 204, the selected images are converted from RGB colour space to HSVcolour space. According to an embodiment of the invention, theconversion from RGB to HSV may be performed by any known method. At step206, the HSV color images may be split into HSV channels. Further, atstep 208 and step 210 the method computes the average value of luminanceand average value of saturation for both the HSV images respectively. Atstep 212 and step 214, percentage deviation of average luminance andaverage saturation may be calculated respectively for the first sampleimage with respect to the second reference image. At step 216 and step218, the percentage deviation of average luminance and percentagedeviation of average saturation is compared with a predefined thresholdvalue respectively. According to an embodiment, the threshold value ismore than 2 percent deviation. According to another embodiment,threshold value may be more than 5 percent deviation. If the percentagedeviation of average luminance is more than the predefined thresholdvalue then the first sample image may be multiplied by a luminancecorrection factor at step 220, else the image may be retained withoutincorporating any change. The luminance correction factor is the ratioof the average value of the illumination of the first sample imagedivided by average value of illumination for the second reference image.Similarly, if the percentage deviation of average saturation is morethan the predefined threshold value, then the first sample image may bemultiplied by a saturation correction factor at step 222, else the imagemay be retained without incorporating any change. The luminancecorrection factor is the ratio of the average value of the saturation ofthe first sample image divided by average value of saturation for thesecond reference image. At step 224, the HSV channels of the processedimages are merged together. The disclosed process may be repeated forall the images in the aligned image stack considering the correctedfirst sample image as second reference image for the next image andsimilarly repeating the process for other images. Once corrected, theimages may be again converted in RGB color space at step 226 and storedin the image stack.

At step 108, the method computes energy content for each pixel ofilluminated and color corrected stacked images to generate energy matrixof each image. According to an exemplary embodiment, complex waveletdecomposition method may be used for wavelet decomposition. According tothe complex wavelet decomposition method, the step of computing energycontent includes selecting one of the images from the illuminated andcolor corrected image stack. Selected image is converted from RGB colorscale to grayscale for wavelet decomposition. The method furtherincludes down sampling the grayscale image to a lower resolutionexemplary by one level and normalizing the intensity values in the rangeof 0 to 1. Processing the image at a lower resolution may reduce theimpulse noises present in the images and hence may provide betterresults. The method further includes, convolving the down sampled imagewith a complex wavelet filter bank to generate an energy matrix for saidimage. The process may be repeated for all the images in the illuminatedand color corrected image stack so as to have at least one energy matrixfor each image in the stack. According to another embodiment the energymatrix may be generated using any other known process such as but notlimited to real wavelets (Haar, Daubechies), difference of Gaussians,variance, Tenengrad, Fourier transform and high pass filter.

At step 110, the method generates a raw index map for the scene. Theprocess of generating raw index map includes analyzing the energymatrix's pixel by pixel basis for all the images and identifying maximumfocused pixel for a particular pixel in the image stack. The process isrepeated for all the pixels of the scene and an index of all the focusedpixels may be used to generate the raw index map.

At step 112, the method generates degree of defocus map by comparing theenergy content at a particular pixel against a reference signal, wherethe reference signal is a Gaussian curve. The Gaussian curve may begenerated by identifying the peak focus measure values and the value ofminima at left side and right side of the peak value for a particularpixel in the image stack by analyzing the energy matrix's generated atstep 108. Using the log of the energy values of maxima and minima theGaussian curve may be generated. The generated Gaussian curve may beused as the reference signal to compare the focus and out of focusregions in different image. The computed result may be used to generatedegree of defocus map. Focus measure values in the regions where theobject is focused, will follow Gaussian nature while other parts havingextensive smoothness in texture follow random nature. Therefore pixelscorresponding to focused part yield low gauss-fit values whereas smoothpixels yield high values. So, the in-focus segmentation of the objectmay be identified using degree of defocus map.

At step 114, the method may generate a composite image using the rawindex map and the degree of defocus map. The index map, which isconstructed by taking the index of the stack corresponding to thehighest frequency in the temporal direction for each pixel, may containnoise (random index) wherever the scene is out of Focus. The noise needsto be removed and the index map needs to be further processed. Accordingto an embodiment, the steps of refining the index map includeseliminating noise by masking the index map with the degree of defocusmap, expanding the masked index map, blurring the result by a smoothingfilter and overlapping the masked index map on blurred output. The outof focus regions may have high measure in the degree of defocus map. Athreshold of at least 25% of the maximum value may be applied to removethe out of focus regions. According to yet another embodiment, in thecomposite image, the out of focus regions may be picked from the lastimage of the stack to avoid blotchiness. The current index values in theout of focus regions may be changed to maximum index values (index ofthe last image). The in-focus region in the index map may be dilated andblurred in order to get a smooth transition between the object indexvalues and the highest index value of the out of focus region. Finally,the masked index map may be placed on the processed image to get therefined index map.

At step 116, the method generates a depth map using the raw index mapand the degree of defocus map. The step of generating a depth mapincludes, performing depth interpolation by polynomial fitting of themaximum index value in the index map. Scaling the image in the availableintensity range for detailed information of depth of object. Subtractingthe background with help of weighted background mask and smooth theimage for noise cancellation and up sampling the generated Depth Map tofit the original size of input image.

According to yet another embodiment, the plurality of source images ofthe scene may be distributed in two or more image stacks. Images in thefirst stack may be processed to generate a composite image by the method100 illustrated above. The generated composite image is then included inthe next image stack and again another composite is generated byprocessing images in the stack by the method 100 illustrated above. Thesame process may be followed for all the stacks to generate a finalcomposite image. Illustrated process may save memory requirement, if theinitial stack contains large number of images.

According to an embodiment, the disclosed exemplary method may beimplemented as a computer program embedded in a carrier, such as but notlimited to, a diskette, a CD-ROM or a modulated carrier wave.

According to yet another embodiment, a system 300 for constructing acomposite image with extended depth of field, from a plurality of sourceimages of a scene is disclosed. The composite image may be constructedfrom the plurality of source images of a scene stored in at least oneimage stack provided in a memory 302. The plurality of source images maybe taken at substantially identical fields of view. According to anembodiment, the system may have arrangement for obtaining images suchthat each image being obtained at a different focal distance. Asillustrated system 300 includes an optical system 304 having a field ofview focused on the object 306. A drive mechanism 308, which may becontrolled by an imaging control 310, may be coupled to the opticalsystem 304 for changing the distance between the optical system 304 andthe object 306. Accordingly the lens may be placed in a succession ofdifferent distances from the object, while concurrently maintaining thesame field of view. According to another embodiment, the image plane maybe kept at a substantially constant distance from the lens, while thefocal length of the optical system 304 may be changed successively. Thecaptured images may be stored in the memory 302.

The disclosed system 300 further includes a processing unit 312.According to an embodiment, the processing unit 312 may process theimages to align the images in the image stack such that every image inthe image stack is aligned with other images in the stack, performillumination and color correction on the aligned images in the imagestack, generate an energy matrix for each pixel of each illumination andcolor corrected image in the image stack by computing energy content foreach pixel, generate a raw index map that contains the location of everypixels having maximum energy level among all the images in the imagestack, generate degree of defocus map by comparing the energy content ata particular pixel against a reference signal and constructing thecomposite image using raw index map and degree of defocus map. Theconstructed image may be stored in the memory 302 or may be sent to anoutput unit 314 such as but not limited to image display, printer, videodisplay screen etc.

In the foregoing detailed description of embodiments of the invention,various features are grouped together in a single embodiment for thepurpose of streamlining the disclosure. This method of disclosure is notto be interpreted as reflecting an intention that the claimedembodiments of the invention require more features than are expresslyrecited in each claim. Rather, as the following claims reflect,inventive subject matter lies in less than all features of a singledisclosed embodiment. Thus, the following claims are hereby incorporatedinto the detailed description of embodiments of the invention, with eachclaim standing on its own as a separate embodiment.

It is understood that the above description is intended to beillustrative, and not restrictive. It is intended to cover allalternatives, modifications and equivalents as may be included withinthe spirit and scope of the invention as defined in the appended claims.Many other embodiments will be apparent to those of skill in the artupon reviewing the above description. The scope of the invention should,therefore, be determined with reference to the appended claims alongwith the full scope of equivalents to which such claims are entitled. Inthe appended claims, the terms “including” and “in which” are used asthe plain-English equivalents of the respective terms “comprising” and“wherein,” respectively.

We claim:
 1. An image processing method for constructing a compositeimage with extended depth of field, from a plurality of source images ofa scene stored in at least one image stack, the plurality of sourceimages being taken at substantially identical fields of view, the methodcomprising: aligning the images in the image stack such that every imagein the image stack is aligned with other images in the stack, whereinthe images are aligned by a bi-directional image alignment method, thebi-directional image alignment method includes, arranging the images ina sequence in the image stack; selecting a first reference image, suchthat the first reference image is the substantially central image in theimage sequence; aligning the first reference image with immediate leftside and immediate right side images in the sequence; and repeating thealignment process for the next image on left side and right side of thereference image with respect to the aligned images; performingillumination and color correction on the aligned images in the imagestack; generating an energy matrix for each pixel of each illuminationand color corrected image in the image stack by computing energy contentfor each pixel; generating a raw index map that contains the location ofevery pixels having maximum energy level among all the images in theimage stack; generating degree of defocus map by comparing the energycontent at a particular pixel in all the images against a referencesignal and repeating the process for all the pixels; and constructingthe composited image using the raw index map and the degree of defocusmap.
 2. The method as claimed in claim 1, wherein the left side andright side images are aligned by a hierarchical coarse-to-fineparametric image alignment process.
 3. The method as claimed in claim 2,wherein the hierarchical coarse-to-fine parametric image alignmentprocess is implemented by a guide optimization process.
 4. The method asclaimed in claim 1, wherein the step of performing illumination andcolor correction comprises: selecting a second reference image and asample image from the aligned images such that the second referenceimage and the sample image are consecutive images in the image stack;converting the selected images from RGB colour space to HSV colourspace; computing an average luminance and an average saturation value ofthe HSV colour space images; calculating the percentage deviation of theaverage luminance and the average saturation for the first sample imagewith respect to the second reference image; multiplying the first sampleimage with a luminance correction factor, if the percentage deviation ofthe average luminance is more than a predefined threshold level, and/ormultiplying the first sample image with a saturation correction factor,if the percentage deviation of the average saturation variation is morethan a predefined threshold level to obtain the modified first sampleimage, else using the first sample image as modified first sample image;repeating the process for all the images, by considering the modifiedfirst sample image image as the first reference image and nextconsecutive image as first sample image; and converting the modifiedimages from HSV color scale to RGB color scale.
 5. The method as claimedin claim 4, wherein the predefined threshold level is more than 2percent deviation of the average luminance and/or the average saturationvariation.
 6. The method as claimed in claim 4, wherein the luminancecorrection factor is the ratio of the average value of the illuminationof the first sample image divided by average value of illumination forthe second reference image.
 7. The method as claimed in claim 4, whereinthe saturation correction factor is the ratio of the average value ofthe saturation of the first sample image divided by average value ofsaturation for the second reference image.
 8. The method as claimed inclaim 1, wherein the step of generating an energy matrix comprises:converting each color and illumination corrected RGB image to grayscaleimage; down sampling the image to low resolution of intensity values inthe range of 0 to 1; and convolving the down sampled image with acomplex wavelet filter bank to generate the energy matrix.
 9. The methodas claimed in claim 1, wherein the reference signal is a Gaussian curveis generated by: taking an array of energy values from energy eachmatrices in temporal direction for each pixel location and identifying apeak value in each array; identifying a local minima on both sides ofthe Peak energy, value, in the array; and generating the Gaussian curveby taking log of the energy values between the maxima and two minima.